Introduction

• “Big Data” is becoming a focal discussion in most of the discipline
• Massive explosion of data with informations integrated in many variables and features
• New methods and algoriths are being devised inorder to extract such information and study the relationship between different variables
• Modern inter-disciplinary research fields such as chemometrics, echonometrics and bioinformatics are handling multi-response models extensively
• This paper attempts to compare some of such methods and their performance on linear model data with specifically designed properties

Statistical Model

• Simulation model \[ \begin{bmatrix} \mathbf{y} \\ \mathbf{x} \end{bmatrix} \sim \mathbf{N}\left( \begin{bmatrix} \boldsymbol{\mu}_{y} \\ \boldsymbol{\mu}_x \end{bmatrix}, \begin{bmatrix} \boldsymbol{\Sigma}_{yy} & \boldsymbol{\Sigma}_{yx} \\ \boldsymbol{\Sigma}_{xy} & \boldsymbol{\Sigma}_{xx} \end{bmatrix} \right) \]
• Define transformation as \(\mathbf{z} = \mathbf{Rx}\) and \(\mathbf{w} = \mathbf{Qy}\)

• Equivalent latent model will be, \[ \begin{bmatrix} \mathbf{w} \\ \mathbf{z} \end{bmatrix} \sim \mathbf{N}\left( \begin{bmatrix} \boldsymbol{\mu}_{w} \\ \boldsymbol{\mu}_x \end{bmatrix}, \begin{bmatrix} \boldsymbol{\Sigma}_{ww} & \boldsymbol{\Sigma}_{wz} \\ \boldsymbol{\Sigma}_{zw} & \boldsymbol{\Sigma}_{zz} \end{bmatrix} \right) \]

• How much should I discuss about simrel-M??

Exerimental Design

• Parameters with single level:
  • Number of observations (n): 100
  • Number of response variables (m): 4
  • Number of informative response components: 1
  • Position of predictor components relevant for response components (relpos): 1, 2, 3, 4
  • Something smart (ypos): 1, 2, 3, 4
• Parameters with multiple level:
  • Number of predictor variables (p): 2 levels (20, 250)
  • Decay factor of eigenvalues corresponding to predictors (gamma): 2 levels (0.2, 0.9)
  • Decay factor of eigenvalues corresponding to response (eta): 2 levels (0, 0.4, 0.8, 1.2)
  • Coefficient of determination corresponding to each informative response compnentsR2: 2 levels (0.8; 0.4)

• Covariance structure of first design is,

  

Estimation Methods

• Methods used in the study and their short description (how they estimate, what are they based on)

  1. Principal Component Regression (PCR)
  2. Partial Least Squares 1 (PLS1)
  3. Partial Least Squares 2 (PLS2)
  4. Envelope Estimation in Predictor Space (Xenv)
  5. Simulteneous envelope estimaion (Senv)
  6. Ridge Regression (Ridge)
  7. Lasso Regression (Lasso)
• How details should I discuss about these methods in terms their way of estimation and difference between them

• As Xenv, Yenv and Senv are based on maximum likelihood estimation, principal components of predictors explaining 99.5% of their variation are used.

Exploratory Study

• This section explores the inter-connection between the estimation methods and the properties of data based on regression coefficients
• Our discussion revolve around following factors and their interaction
  • Wide vs Tall predictor matrix
  • High vs Low multicollinearity
  • High vs Low correlation between responses
  • Hight vs Low coefficient of determination

Systematic Comparison

• Should we use MANOVA model or some kind of norm/trace or similar measure for the error and use ANVOA instead?
• A MANOVA model is used for statistical analysis \[\mathtt{pred\_err}_{ijklm} = \mu + \mathtt{p}_i * \mathtt{gamma}_j * \mathtt{r2}_k * \mathtt{method}_l * \mathtt{eta}_m + \epsilon_{ijklm}\]

• In the model the prediction error for each of three response variables are used as response variable and following variables (with levels) and their complete interactions are used as predictor variables.

  1. Number of predictor variables \((p)\): 20 and 250
  2. Decay factor of eigenvalues of \(X\) \((\gamma)\): 0.2 and 0.9
  3. Decay factor of eigenvalues of \(Y\) \((\eta)\): 0 and 0.4 and 0.8 and 1.2
  4. Coefficient of Determination \((\rho)\): 0.8 and 0.4
  5. Method of estimation: PCR, PLS1, PLS2, Xenvelope, Yenvelope, Senvelope, Ridge and Lasso
  6. Number of tuning Parameters used (as numeric)

• Following is the MANOVA output for estimation error and prediction error models using number of components (tuning parameters) that results minimum error.

  Estimation Error Model:

  Analysis of Variance Table
  
                           Df  Pillai approx F num Df den Df    Pr(>F)    
  (Intercept)               1 0.73310   7535.0      4  10973 < 2.2e-16 ***
  p                         1 0.04149    118.7      4  10973 < 2.2e-16 ***
  gamma                     1 0.52520   3034.4      4  10973 < 2.2e-16 ***
  eta                       3 0.03545     32.8     12  32925 < 2.2e-16 ***
  R2                        1 0.05762    167.7      4  10973 < 2.2e-16 ***
  Method                    6 0.72237    403.2     24  43904 < 2.2e-16 ***
  p:gamma                   1 0.00871     24.1      4  10973 < 2.2e-16 ***
  p:eta                     3 0.00192      1.8     12  32925 0.0490926 *  
  gamma:eta                 3 0.00925      8.5     12  32925 2.631e-16 ***
  p:R2                      1 0.00772     21.4      4  10973 < 2.2e-16 ***
  gamma:R2                  1 0.03540    100.7      4  10973 < 2.2e-16 ***
  eta:R2                    3 0.00145      1.3     12  32925 0.1937604    
  p:Method                  6 0.11772     55.5     24  43904 < 2.2e-16 ***
  gamma:Method              6 0.58455    313.1     24  43904 < 2.2e-16 ***
  eta:Method               18 0.01487      2.3     72  43904 4.387e-09 ***
  R2:Method                 6 0.27841    136.9     24  43904 < 2.2e-16 ***
  p:gamma:eta               3 0.00122      1.1     12  32925 0.3404456    
  p:gamma:R2                1 0.00163      4.5      4  10973 0.0013196 ** 
  p:eta:R2                  3 0.00109      1.0     12  32925 0.4445299    
  gamma:eta:R2              3 0.00089      0.8     12  32925 0.6332008    
  p:gamma:Method            6 0.02259     10.4     24  43904 < 2.2e-16 ***
  p:eta:Method             18 0.00988      1.5     72  43904 0.0033841 ** 
  gamma:eta:Method         18 0.01067      1.6     72  43904 0.0005951 ***
  p:R2:Method               6 0.02578     11.9     24  43904 < 2.2e-16 ***
  gamma:R2:Method           6 0.18226     87.3     24  43904 < 2.2e-16 ***
  eta:R2:Method            18 0.00791      1.2     72  43904 0.1100396    
  p:gamma:eta:R2            3 0.00138      1.3     12  32925 0.2337892    
  p:gamma:eta:Method       18 0.00785      1.2     72  43904 0.1199665    
  p:gamma:R2:Method         6 0.00553      2.5     24  43904 4.908e-05 ***
  p:eta:R2:Method          18 0.00723      1.1     72  43904 0.2551289    
  gamma:eta:R2:Method      18 0.00610      0.9     72  43904 0.6419794    
  p:gamma:eta:R2:Method    18 0.00757      1.2     72  43904 0.1709646    
  Residuals             10976                                             
  ---
  Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

  Prediction Error Model:

  Analysis of Variance Table
  
                           Df  Pillai approx F num Df den Df    Pr(>F)    
  (Intercept)               1 0.99979 13042636      4  10973 < 2.2e-16 ***
  p                         1 0.00031        1      4  10973  0.488646    
  gamma                     1 0.10168      311      4  10973 < 2.2e-16 ***
  eta                       3 1.00003     1372     12  32925 < 2.2e-16 ***
  R2                        1 0.98330   161522      4  10973 < 2.2e-16 ***
  Method                    6 0.20035       96     24  43904 < 2.2e-16 ***
  p:gamma                   1 0.00019        1      4  10973  0.713312    
  p:eta                     3 0.00112        1     12  32925  0.420803    
  gamma:eta                 3 0.00448        4     12  32925 1.937e-06 ***
  p:R2                      1 0.00036        1      4  10973  0.410451    
  gamma:R2                  1 0.00107        3      4  10973  0.018992 *  
  eta:R2                    3 0.00810        7     12  32925 7.492e-14 ***
  p:Method                  6 0.00460        2     24  43904  0.001231 ** 
  gamma:Method              6 0.09142       43     24  43904 < 2.2e-16 ***
  eta:Method               18 0.07152       11     72  43904 < 2.2e-16 ***
  R2:Method                 6 0.04785       22     24  43904 < 2.2e-16 ***
  p:gamma:eta               3 0.00143        1     12  32925  0.207494    
  p:gamma:R2                1 0.00120        3      4  10973  0.010454 *  
  p:eta:R2                  3 0.00174        2     12  32925  0.087104 .  
  gamma:eta:R2              3 0.00127        1     12  32925  0.303652    
  p:gamma:Method            6 0.00275        1     24  43904  0.178302    
  p:eta:Method             18 0.00857        1     72  43904  0.040237 *  
  gamma:eta:Method         18 0.02426        4     72  43904 < 2.2e-16 ***
  p:R2:Method               6 0.00313        1     24  43904  0.079073 .  
  gamma:R2:Method           6 0.00283        1     24  43904  0.150836    
  eta:R2:Method            18 0.01680        3     72  43904 7.002e-12 ***
  p:gamma:eta:R2            3 0.00248        2     12  32925  0.007171 ** 
  p:gamma:eta:Method       18 0.00356        1     72  43904  0.999463    
  p:gamma:R2:Method         6 0.00196        1     24  43904  0.605334    
  p:eta:R2:Method          18 0.00716        1     72  43904  0.275495    
  gamma:eta:R2:Method      18 0.00555        1     72  43904  0.817756    
  p:gamma:eta:R2:Method    18 0.00645        1     72  43904  0.512952    
  Residuals             10976                                             
  ---
  Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

• Study of Effect

  

Discussion and Conclusion